The present invention relates to signal processing apparatus such as a neural computer modeled on a neural network, which apparatus is applicable to the recognition of characters and drawings, the motion control of robots or the like, and memory by association.
An artificial neural network is constructed such that functions of neurons which are basic units of information processing in a living body are simulated in the form of elements resembling neurons, and such that these elements are put together in a network so that parallel information processing can be carried out. In a living body, processes such as character recognition, memory by association and control of motion can be carried out quite easily. However, such processes are often extremely difficult to carry out on Neumann computers. In order to cope with these problems, attempts are rigorously carried out for simulating functions characteristic of a living body, that is, parallel processing and self learning, which are effected by a nervous system of a living body. However, such attempts are in many cases realized by computer simulations. In order to bring out the advantageous features of the neural network, it is necessary to realize the parallel processing by hardware. Some proposals have been made to realize the neural network by hardware, however, the proposed neural networks cannot realize the self learning function which is an advantageous feature of the neural network. Further, the majority of the proposed networks are realized by analog circuits and has a problem of unfavorable operation.
A detailed discussion of the above points follows. First, a description will be given of a model of a conventional neural network. FIG. 1 shows one neuron unit (an element resembling a neuron) 1, and FIG. 2 shows a neural network which is made up of a plurality of such neuron units 1. Each neuron unit 1 of the neural network is coupled to and receives signals from a plurality of neuron units 1, and outputs a signal by processing the received signals. In FIG. 2, the neural network has a hierarchical structure, and each neuron unit 1 receives signals from the neuron units 1 located in a previous layer shown on the left side and outputs a signal to the neuron units 1 located in a next layer shown on the right side.
In FIG. 1, T.sub.ij denotes a weight function which indicates the intensity of coupling (or weighting) between an ith neuron unit and a jth neuron unit. The coupling between first and second neuron units is referred to as an excitatory coupling when a signal output from the second neuron unit increases as a signal received from the first neuron unit increases. On the other hand, the coupling between the first and second neuron units is referred to as an inhibitory coupling when the signal output from the second neuron decreases as the signal received from the first neuron unit increases. T.sub.ij &gt;0 indicates the excitatory coupling, and T.sub.ij &lt;0 indicates the inhibitory coupling. When an output signal of the ith neuron unit 1 is denoted by y.sub.i, the input signal to the jth neuron unit 1 from the ith neuron unit 1 can be described by T.sub.ij y.sub.i. Since a plurality of neuron units 1 are coupled to the jth neuron unit 1, the input signals to the jth neuron unit 1 can be described by .SIGMA.T.sub.ij y.sub.i. The input signals .SIGMA.T.sub.ij y.sub.i to the jth neuron unit 1 will hereinafter be referred to as an internal potential u.sub.j of the jth neuron unit 1 as defined by the following equation (1). EQU u.sub.j =.SIGMA.T.sub.ij Y.sub.i ( 1)
Next, it will be assumed that a non-linear process is carried out on the input. The non-linear process is described by a non-linear neuron response function using a sigmoid function as shown in FIG. 3 and the following equation (2). EQU f(x)=1/(1+e.sup.-x) (2)
Hence, in the case of the neural network shown in FIG. 2, the equations (1) and (2) are successively calculated for each weight function T.sub.ij so as to obtain a final result.
FIG. 4 shows an example of a conventional neuron unit proposed in a Japanese Laid-Open Patent Application No.62-295188. The neuron unit includes a plurality of amplifiers 2 having an S-curve transfer function, and resistive feedback circuit network 3 which couples outputs of each of the amplifiers 2 to inputs of amplifiers in another layer as indicated by a one-dot chain line. A time constant circuit 4 made up of a grounded capacitor and a grounded resistor is coupled to an input of each of the amplifiers 2. Input currents I.sub.1, I.sub.2, . . . , I.sub.N are respectively supplied to the inputs of the amplifiers 2, and the output is derived from a collection of output voltages of the amplifiers 2.
An intensity of the input signal and the output signal is described by a voltage, and an intensity of the coupling (or weighting) between the neuron units is described by a resistance of a resistor 5 (a lattice point within the resistive feedback circuit network 3) which couples the input and output lines of the neuron units. A neuron response function is described by the transfer function of each amplifier 2. In addition, the coupling between the neuron units may be categorized into the excitatory and inhibitory couplings, and such couplings are mathematically described by positive and negative sings on weight functions. However, it is difficult to realize the positive and negative values by the circuit constants. Hence, the output of the amplifier 2 is distributed into two signals, and one of the two signals is inverted so as to generate a positive signal and a negative signal. One of the positive and negative signals derived from each amplifier 2 is appropriately selected. Furthermore, an amplifier is used to realize the sigmoid function shown in FIG. 3.
However, the above described neuron unit suffers from the following problems.
1) The weight function T.sub.ij is fixed. Hence, a value which is learned beforehand through a simulation or the like must be used for the weight function T.sub.ij, and a self-learning cannot be made.
2) When the accuracy and stability of one neuron unit are uncertain, new problems may arise when a plurality of such neuron units are used to form the neural network. As a result, the operation of the neural network becomes unpredictable.
3) When the neural network is formed by a large number of neuron units, it is difficult to obtain the large number of neuron units which have the same characteristic.
4) Because the signal intensity is described by an analog value of the potential or current and internal operations are also carried out in the analog form, the output value easily changes due to the temperature characteristic, the drift which occurs immediately after the power source is turned ON and the like.
On the other hand, as a learning rule used in numerical calculations, there is a method called back propagation which will be described hereunder.
First, the weight functions are initially set at random. When an input is supplied to the neural network in this state, the resulting output is not necessarily a desirable output. For example, in the case of character recognition, a resulting output "the character is `L`" is the desirable output when a handwritten character "L" is the input, however, this desirable output is not necessarily obtained when the weight functions are initially set at random. Hence, a correct solution (teaching signal) is input to the neural network and the weight functions are varied so that the correct solution is output when the input is the same. The algorithm for obtaining the varying quantity of the weight functions is called the back propagation.
For example, in the hierarchical neural network shown in FIG. 2, the weight function T.sub.ij is varied using the equation (4) so that E described by the equation (3) becomes a minimum when the output of the jth neuron unit in the output (last) layer is denoted by y.sub.j and the teaching signal with respect to this jth neuron unit is denoted by d.sub.j. EQU E=.SIGMA.(d.sub.j -y.sub.j).sup.2 ( 3) EQU .DELTA.T.sub.ij =.delta.E/.delta.T.sub.ij ( 4)
Particularly, when obtaining the weight functions of the output layer and the layer immediately preceding the output layer, an error signal .delta. is obtained using the equation (5), where f' denotes a first order differential function of the sigmoid function f. EQU .delta..sub.j =(d.sub.j -y.sub.j).times.f'(u.sub.j) (5)
When obtaining the weight functions of the layers preceding the layer which immediately precedes the output layer, the error signal .delta. is obtained using the equation (6). EQU .delta..sub.j .SIGMA..delta..sub.j T.sub.ij .times.f'(u.sub.j)(6)
The weight function T.sub.ij is obtained from the equation (7) and varied, where T.sub.ij ' and T.sub.ij ' are values respectively obtained during the previous learning, .eta. denotes a learning constant and .alpha. denotes a stabilization constant. EQU .DELTA.T.sub.ij =(.delta..sub.j y.sub.i)+.alpha.T.sub.ij ' EQU T.sub..sub.ij =T.sub.ij '+T.sub.ij ( 7)
The constants .eta. and .alpha. are obtained through experience since these constants .eta. and .alpha. cannot be obtained logically.
The neural network learns in the above described manner, and an input is thereafter applied again to the neural network to calculate an output and learn. By repeating such an operation, the weight function T.sub.ij is determined such that a desirable resulting output is obtained for a given input.
When an attempt is made to realize the above described learning function, it is extremely difficult to realize the learning function by a hardware structure since the learning involves many calculations with the four fundamental rules of arithmetics.
On the other hand, a neural network realized by digital circuits has been proposed. A description will now be given, with reference to FIGS. 5 through 7, of such digital neural networks. FIG. 5 shows a circuit construction of a single neuron. In FIG. 5, each synapse circuit 6 is coupled to a cell circuit 8 via a dendrite circuit 7. FIG. 6 shows an example of the synapse circuit 6. In FIG. 6, a coefficient multiplier circuit 9 multiplies a coefficient a to an input pulse f, where the coefficient a is "1" or "2" depending on the amplification of a feedback signal. A rate multiplier 10 receives an output of the coefficient multiplier circuit 9. A synapse weighting register 11 which stores a weight function w is connected to the rate multiplier 10. FIG. 7 shows an example of the cell circuit 8. In FIG. 7, a control circuit 12, an up/down counter 13, a rate multiplier 14 and a gate 15 are successively connected in series. In addition, an up/down memory 16 is connected as shown.
In this proposed neural network, the input and output of the neuron circuit is described by a pulse train, and the signal quantity is described by the pulse density of the pulse train. The weight function is described by a binary number and stored in the memory 16. The input signal is applied to the rate multiplier 14 as the clock and the weight function is applied to the rate multiplier 14 as the rate value, so that the pulse density of the input signal is reduced depending on the rate value. This corresponds to the term T.sub.ij y.sub.i Of the back propagation model. The portion which corresponds to .SIGMA. of .SIGMA.T.sub.ij y.sub.i is realized by an OR circuit which is indicated by the dendrite circuit 7. Because the coupling may be excitatory or inhibitory, the circuit is divided into an excitatory group and an inhibitory group and an OR operation is carried out independently for the excitatory and inhibitory groups. Outputs of the excitatory and inhibitory groups are respectively applied to up-count and down-count terminals of the counter 13 and counted in the counter 13 which produces a binary output. The binary output of the counter 13 is again converted into a corresponding pulse density by use of the rate multiplier 14. A plurality of the neurons described above are connected to form a neural network. The learning of this neural network is realized in the following manner. That is, the final output of the neural network is input to an external computer, a numerical calculation is carried out within the external computer, and a result of the numerical calculation is written into the memory 16 which stores the weight function. Accordingly, this neural network does not have the self-learning function. In addition, the circuit construction of this neural network is complex because a pulse density of a signal is once converted into a numerical value by use of a counter and the numerical value (binary number) is again converted back into a pulse density.
Therefore, the conventional neural network or neural network suffers from the problem in that the self-learning function cannot be realized by hardware. Furthermore, the analog circuits do not provide stable operations, and the learning method using numerical calculation is extremely complex and is unsuited to be realized by hardware. On the other hand, the circuit construction of the digital circuits which provide stable operations is complex.
Attempts to resolve the above-mentioned problems include the proposals disclosed in the Japanese Laid-Open Patent Application No.4-549 (Basics of forward process) which discloses a pulse-density type neuron model capable of self-learning, and the Japanese Laid-Open Patent Application No.4-111185 (Basics of learning process), both of these proposals being made by the present applicants.
When a learning occurs in these neural networks, a differential coefficient, obtained by differentiating an output signal from a neuron with respect to an internal potential, is used to calculate an error signal. A disadvantage of the aforementioned proposals is that this differential coefficient is set to be constant. Therefore, the learning capability is not satisfactorily high, thus leaving much to be desired.